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Abstract 

Background: Providers and policymakers are pursuing strategies to increase patient engagement in health care. 
Increasingly, online sections of medical records are viewable by patients though seldom are clinicians' visit notes 
included. We designed a one-year multi-site trial of online patient accessible office visit notes, OpenNotes. We 
hypothesized that patients and primary care physicians (PCPs) would want it to continue and that OpenNotes 
would not lead to significant disruptions to doctors' practices. 

Methods/Design: Using a mixed methods approach, we designed a quasi-experimental study in 3 diverse 
healthcare systems in Boston, Pennsylvania, and Seattle. Two sites had existing patient internet portals; the third 
used an experimental portal. We targeted 3 key areas where we hypothesized the greatest impacts: beliefs and 
attitudes about OpenNotes, use of the patient internet portals, and patient-doctor communication. PCPs in the 3 
sites were invited to participate in the intervention. Patients who were registered portal users of participating PCPs 
were given access to their PCPs' visit notes for one year. PCPs who declined participation in the intervention and 
their patients served as the comparison groups for the study. We applied the RE-AIM framework to our design in 
order to capture as comprehensive a picture as possible of the impact of OpenNotes. We developed pre- and 
post-intervention surveys for online administration addressing attitudes and experiences based on interviews and 
focus groups with patients and doctors. In addition, we tracked use of the internet portals before and during the 
intervention. 

Results: PCP participation varied from 19% to 87% across the 3 sites; a total of 1 14 PCPs enrolled in the 
intervention with their 22,000 patients who were registered portal users. Approximately 40% of intervention and 
non-intervention patients at the 3 sites responded to the online survey, yielding a total of approximately 38,000 
patient surveys. 

Discussion: Many primary care physicians were willing to participate in this "real world" experiment testing the 
impact of OpenNotes on their patients and their practices. Results from this trial will inform providers, policy 
makers, and patients who contemplate such changes at a time of exploding interest in transparency, patient safety, 
and improving the quality of care. 
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Providers and policymakers are pursuing many strategies 
to increase the engagement of patients in promoting 
health and managing illness. As the general trend 
toward transparency accelerates, providers with electro- 
nic medical records (EMRs) increasingly allow patients 
to view online laboratory results, medication lists, and 
other parts of the medical record. However, while 
patients nationwide have the right to view their medical 
records, providers rarely share clinicians' visit notes 
proactively with their patients. 

Offering visit notes to patients stirs concerns among 
doctors and their staff. Will the many patients inundate 
their doctors' offices with telephone and e-mail queries, 
placing excessive demands on already overburdened 
doctors and their staffs? Will patients be distressed or 
confused by what they read? Or would such access lead 
to more involvement in care, more sense of control, 
greater knowledge, and improved adherence to the plan 
of care? How would patients view comments about their 
mental health, likelihood of developing a malignancy, 
habits with alcohol, or efforts to control their weight? 

Whatever the views of doctors or patients, in the pre- 
sence of rapidly changing societal norms such change 
toward greater transparency will almost certainly occur 
in coming years. On the cusp of this change, we sought 
to begin to answer some important questions and con- 
cerns through the OpenNotes study, using a simple, one 
step intervention. We made all visit notes between con- 
senting primary care physicians (PCPs) and their patients 
readily available online through secure patient Internet 
portals. We hypothesized that after a one year trial of 
OpenNotes, patients and PCPs would want it to con- 
tinue. And we hypothesized further that OpenNotes 
would not lead to significant disruptions for doctors and 
their staffs. 

In designing this trial, which would involve both a 
demonstration and evaluation of OpenNotes, we 
weighed the strategy of a standard experimental design 
against an effort that would focus on a large scale inter- 
vention involving multiple and highly diverse sites, each 
of which required a somewhat different intervention. 
We chose the latter approach in order to mount a "real 
world" experiment involving a large number of patients 
and PCPs in a one-year demonstration of OpenNotes. 
The many PCPs who declined participation in the inter- 
vention and their patients would thereby serve as the 
comparison groups for the trial. Basing our trial in the 
primary care environment in 3 different settings and 
geographical locations, our primary purpose was to 
assess PCPs and patients attitudes and experiences with 
OpenNotes. Secondly, we sought to gather information 
from a variety of sources in order to determine in a 
comprehensive way how the use of OpenNotes affects 
patient behavior and doctors' practices. 



Crafting a careful evaluation of this intervention, 
which many viewed as potentially disruptive, across very 
different health care systems required both extensive 
planning and creative strategies to address the many dif- 
ferences in the populations and technologies across the 
3 sites. With the hope that our unusual design might 
help inform other studies focusing on the evaluation of 
innovations in the practice of medicine, we describe in 
this paper the methods we developed and implemented. 
Also, we present initial results from our recruitment 
and implementation of OpenNotes. 

Methods/Design 

Settings 

Although we specifically targeted primary care in our eva- 
luation, the 3 health care systems participating in the study 
differed in many ways: Beth Israel Deaconess Medical Cen- 
ter (BIDMC) in Boston is a Harvard teaching hospital with 
approximately 113 hospital-based and community PCPs, 
serving approximately 90,000 patients, including over 
50,000 patients registered to use PatientSite, a secure 
patient Internet portal based on a home-grown electronic 
medical record (EMR). Geisinger Health System (GHS) in 
rural northeastern Pennsylvania has approximately 175 
PCPs serving about 300,000 primary care patients, with 
about 163,000 patients registered to use MyGeisinger, a 
patient portal based on the Epic EMR (Epic Systems Corp., 
Verona, WI). Harborview Medical Center (HMC) is a 
county-owned hospital in Seattle, WA managed by the 
University of Washington; HMC has a specific mission to 
care for the community's most vulnerable patients and is 
also the Disaster Control Hospital for Seattle and King 
County. HMCs two participating primary care clinics are 
staffed by approximately 65 faculty and fellows serving 
approximately 7,000 inner city and indigent patients, 
including about 2,000 patients with HIV. The internally- 
developed HMC patient portal, HealthReach, had been 
used only for research studies and not offered to the gen- 
eral HMC primary care patient population at the time we 
began this project. HealthReach was not integrated with 
HMCs EMR but could display information from ORCA, 
their clinical information system was based on the Cerner 
Millenium EMR platform (Cerner Corp., North Kansas 
City, MO) and other ancillary systems. 

Study design 

In preparation for our OpenNotes demonstration and 
evaluation, we convened a team of investigators from 
across the U.S., including three (SR, JR, and TD) who had 
expertise in patient access to medical records and, speci- 
fically, doctors' visit notes. Informed by the literature and 
our own scientific experience related to patient and doc- 
tor attitudes toward open access to medical records [1-3], 
we developed a set of hypotheses about the potential 
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impact of OpenNotes on primary care doctors and their 
patients, described in detail elsewhere [4], In designing 
our test of the intervention, we specified 3 key areas 
where we hypothesized the greatest impacts would occur: 
beliefs and attitudes about patient accessible notes, use of 
the patient internet portals, and patient-doctor commu- 
nication. We developed a quasi-experimental design that 
would allow us to perform a number of comparisons, 
portrayed in Figure 1, comparing doctors and patients, 
participants and non-participants, before, during, and 
after the one-year experience with OpenNotes. 

We employed mixed methods, using qualitative data 
from semi-structured interviews and focus groups to 
inform survey instrument development and, at the end of 
the study, to provide a more in-depth assessment of 
patients' and doctors' experiences with OpenNotes. 
Using Creswell's typology, this approach is described as a 
sequential study involving qualitative data collection for 
instrument development, followed by quantitative data 
collection and subsequent qualitative data collection for 
triangulation purposes [5]. To structure our evaluation, 
we applied the RE-AIM framework [6] in order to cap- 
ture as comprehensive a picture as possible of the impact 
of OpenNotes on patients, doctors, and health care sys- 
tems. A scientific approach to assessing both the internal 
and external validity of real-world interventions, RE- 
AIM, is an acronym for: Reach, Effectiveness, Adoption, 
Implementation, and Maintenance. To address each of 
these elements of the intervention, we drew upon multi- 
ple data sources, including PCP and patient surveys, 
administrative and billing records, portal utilization data 
and electronic health records. The framework for our 
data collection and analysis is summarized in Table 1, 
Analyses directed to our primary and secondary study 
hypotheses will involve largely quantitative data from 



pre- and post-surveys of both doctors and patients, and 
also patient and doctor use of the Internet portals; subse- 
quent qualitative analyses for secondary aims will utilize 
data obtained from patients' narrative survey information 
and patient and PCP focus groups. 

We considered several study designs, but given the 
scope of the study and the involvement of 3 health sys- 
tems, we identified a quasi-experimental, nonequivalent 
group, pretest-posttest approach as the one that would 
be most informative and efficient [7]. It was not feasible 
to use a traditional experimental approach because we 
knew that many providers would hesitate to be part of 
the first wave of participants for an intervention that 
would potentially impose further burdens on their 
already stretched schedules. Randomized allocation on 
the patient or provider level would have limited the study 
to only those PCPs who agreed to have their visit notes 
posted online and their patients and would have resulted 
in a much smaller number of patients who had access to 
their notes during the intervention period. Maximizing 
participation by allowing all willing PCPs to offer Open- 
Notes to their patient panels provided for the first large- 
scale effort to assess the impact of sharing visit notes 
electronically with patients. 

Survey instrument development 

After developing our own list of expectations about the 
impact of OpenNotes [4] and before the one-year study 
began, we turned to patients and doctors in a series of 
focus groups to further explore attitudes and expecta- 
tions about the intervention. The PCP focus group guide 
addressed communication between patients and PCPs, 
expected impact on PCP practice and workload, and 
impact on patients. Similarly, the patient focus group 
guide dealt with current communication and means of 



Participating PCPs 
Baseline, Follow-up 



I 



Participating Patients 
Baseline, Follow-up 



Non-Participating PCPs 
Baseline 



Non-Participating Patients 
Baseline, Follow-up 



Figure 1 Conceptual model showing primary planned comparisons among doctors and patients in their attitudes, experiences, and 
portal use before and after the implementation of OpenNotes. Primary planned comparisons are indicated by red arrows. 
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Table 1 Summary of RE-AIM framework [6] for Evaluation of OpenNotes 



Domain 


Methods 


Outcomes 


Reach 


Administrative data re: PCP characteristics and panel 
information; Portal use data; Surveys of patients/PCPs 


PCP and patient demographics PCP workload, sessions, visits PCP portal 
statistics: number patients on portal, number of OpenNotes Patient 
portal statistics: registered users, logons 


Effectiveness 


PCP pre- and post- intervention survey, interviews 
Patient pre- and post- intervention survey PCP/patient 
portal data 


PCP and patient preferences about continuation of OpenNotes PCP 
reported burden from OpenNotes Patient measures: Perceptions of 
Benefits/Risks of OpenNotes, ACES Quality of patient-doctor relationship 
subscale, Perceived Efficacy in Patient-Physician Interactions (PEPPI) short 
form questionnaire, PCP portal and telephone messages Patient portal: 
number of notes viewed 


Adoption 


System level data PCP practice data 


Description of systems, PCP practices, proportions using OpenNotes PCP 
panel size, registered portal users 


implementation 


Process data, PCP surveys and interviews, non- 
respondent surveys 


PCP expectations, burden, barriers to participation; detailed descriptions 
of PCP experiences, comments, recommendations 


Maintenance 


PCP/system responses 


Decision to continue or discontinue OpenNotes 



understanding regarding plan of care, and experiences 
and expectations about reading doctor's visit notes. Fol- 
lowing 2 PCP focus groups (6 PCPs at GHS, and 8 at 
BIDMC) and 5 patient focus groups (11 patients at 
BIDMC, and 30 patients at HMC), three investigators 
(TD, EV, JW) reviewed transcripts of the recorded group 
sessions and identified common themes. 

We developed survey questions based on the themes of 
PCP expectations about the impact on their practice and 
PCP and patient attitudes about benefits and risks of 
OpenNotes. Other than the set of questions specifically 
for PCPs about potential practice effects of OpenNotes, 
the PCP and patient surveys were designed in parallel to 
address similar questions about the anticipated benefits 
and risks of OpenNotes. In planning ahead for the one- 
year follow-up survey, we decided we would also include 
a set of questions about attitudes toward possible future 
modifications to OpenNotes that might further engage 
patients and their families in their care. We performed a 
series of tests of the pre-intervention surveys, first using 
paper questionnaires to ascertain the intent and clarity of 
the questions, then incorporating changes based on feed- 
back from PCPs and patients, and we conducted addi- 
tional testing in the online format to assess content and 
potential technical issues. 

Concordant with the study's aims regarding the impact 
on attitudes, we developed 3 items to directly query 
patient respondents about whether or not they: a) thought 
OpenNotes was a good idea, b) would like to read their 
notes, and c) would be likely to share their notes with 
others. To explore these attitudes further, we developed 
three item sets, Perceptions of Benefits and Perceptions of 
Risks of OpenNotes on Patients, comprising 7 and 4 
items, respectively, and a set of 4 items about Risks to 
PCP practices. 

The survey questions regarding perceived patient ben- 
efits were as follows: 



1. I would better understand my health and medical 
conditions. 

2. I would better remember the plan for my care. 

3. I would take better care of myself. 

4. I would be more likely to take my medications as 
prescribed. 

5. I would feel more in control of my health care. 

6. I would be better prepared for visits. 

7. Patients will trust me more as their doctor. (PCP 
survey only) 



The questions addressing perceived patient risks 
as follows: 



were 



1. I would worry more 

2. I would be concerned about my privacy. 

3. The notes would be more confusing than helpful. 

4. It could make my doctor's job more difficult. 
(Patient survey only) 

The PCP survey questions about the impact on their 
practice were as follows: 

1. Patients will disagree with what I write in their 
visit notes 

2. Patients will request changes to the content of 
visit notes 

3. Patients will find significant errors in the notes 

4. Patients will contact me or my practice with ques- 
tions about their notes 

The patient benefits and risk items were designed as 
parallel items on both patient and PCP surveys except 
where otherwise indicated. Response options were as 
follows: agree, somewhat agree, somewhat disagree, dis- 
agree; we added a "don't know" response option for 
patients. To compare PCP perceptions about anticipated 
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and experienced impact of OpenNotes on their practices 
in the pre- and post-intervention surveys, respectively, 
we designed a set of questions addressing PCP docu- 
mentation (content and process), communications with 
patients, potential impact on medical liability, and 
effects on patient care. In the baseline survey, we also 
included items about doctor and practice characteristics. 
For the post-intervention survey, to address our primary 
hypothesis, doctors and their patients in the intervention 
were specifically asked whether or not they would like 
OpenNotes to continue. 

To measure patients' perceptions about patient-doctor 
communication, we used 2 standardized instruments. The 
Ambulatory Care Experiences Survey (ACES) was devel- 
oped to survey patients about their experiences with 
health care providers, as a complement to existing health 
plan level strategies for assessing health care quality [8] . 
We used the 6-item ACES subscale assessing the quality 
of patient-provider interactions. Patients' self-confidence 
for communicating with doctors was measured using the 
validated Perceived Efficacy in Patient- Physician Interac- 
tions (PEPPI) scale [9] . This instrument was based on the 
previous work of Bandura regarding the assessment of 
self-efficacy, a motivator for health behavior [10,11]. The 
survey also included a set of questions regarding Internet 
use and sociodemographic and health characteristics (self- 
rated general health, education, race/ethnicity, and 
employment status). Lastly, in the follow-up survey, we 
included several open-ended questions, giving respondents 
opportunities to describe their experiences with Open- 
Notes and, if they did not read their notes, their reasons 
for not doing so. 

For both patients and doctors, surveys were conducted 
online using SurveyGizmo (v2.0 & v3.0. Widgix, LLC dba 
SurveyGizmo, Boulder, Colorado). We required a response 
for all items except for demographic items, free text ques- 
tions and questions within skip patterns. Based on our 
pilot tests, the PCP surveys were designed to be completed 
in less than 10 minutes, and the patient surveys in less 
than 20 minutes. The instruments are available from the 
authors upon request. Results regarding PCP and patients 
attitudes from the baseline survey were published else- 
where [12]. 

Human subjects protections 

Before the start of the study, all study protocols, data 
collection procedures and the intervention methods 
were approved by the institutional review boards of the 
3 participating institutions, the BIDMC, the GHS and 
the University of Washington. 

Recruitment of primary care doctors 

We began the study with the recruitment and enroll- 
ment of primary care doctors who would be willing to 



make their visit notes available to their patients through 
the Internet portal. OpenNotes doctors signed an 
informed consent prior to completing the baseline study 
surveys that preceded the OpenNotes intervention. 

Invitations to join the intervention went to all primary 
care doctors (excluding trainees) practicing in BIDMC 
and affiliated primary care practices who were using both 
electronic health records (EHR) and the patient portal, 
PatientSite. PCPs were informed about the study through 
email announcements, presentations by study investiga- 
tors, and informal discussions. We obtained the support 
of key clinical leaders within the institution who 
announced their endorsements on websites and through 
email communications with the PCPs. Similar recruit- 
ment strategies were employed at HMC and GHS, but 
because the GHS primary care practices are distributed 
geographically across a large area of rural central and 
northeastern Pennsylvania, the bulk of the recruitment 
contacts at GHS took place via email. 

PCPs from the BIDMC and GHS primary care practices 
who chose not to offer any of their patients access to their 
visit notes online, referred to here as "non-intervention 
PCPs," were included in the study as the PCP comparison 
group. Their patients who were registered portal users 
comprised the comparison group referred to as "non- 
intervention patients." As no patient portal was available 
previously to patients at HMC, we did not have PCP or 
patient comparison groups at HMC. 

Recruitment of patients 

After PCPs agreed to join the study, we sent them a list of 
their eligible patients and allowed them to exclude indivi- 
dual patients from the intervention with no requirement 
that they explain the exclusions. At BIDMC and GHS, 
patient eligibility was defined as having been registered on 
the patient Internet portal for at least 1 year before the 
start of the study. Following PCPs' exclusions, all patients 
registered on the patient Internet portals at BIDMC and 
GHS were included automatically as OpenNotes patients 
for the study, regardless of their portal use history or parti- 
cipation in study surveys. Prior to the start of the interven- 
tion, OpenNotes patients were sent a message through the 
portal informing them that their doctor was participating 
in OpenNotes, with an explanation of the study and a link 
to the OpenNotes website http://www.myopennotes.org 
that included detailed information about the study and the 
research team. The message included an invitation and a 
Web link to the pre-intervention survey. 

We included two comparison groups of patients in our 
evaluation from BIDMC and GHS. The first group com- 
prised patients of non-intervention PCPs who were regis- 
tered portal users, referred to as "non-intervention 
patients." The second group included all patients of 
OpenNotes PCPs who were not registered portal users. 



Leveille et al. BMC Medical Informatics and Decision Making 2012, 12:32 
http://www.biomedcentral.eom/1 472-6947/1 2/32 



Page 6 of 10 



This latter group was not approached to participate in 
surveys; their health care utilization and administrative 
data will be used in the evaluation. As noted above, there 
was no comparison group of HMC patients. 

Prior to the OpenNotes Study, patients at HMC did not 
have access to the patient internet portal which was devel- 
oped and implemented previously for research studies 
within the University of Washington system [2,13]. There- 
fore, the research team actively recruited HMC patients, 
many of whom were indigent or homeless but used the 
Internet through community venues. Eligibility required 
that a patient could communicate in English and had a 
current e-mail address. Some patients obtained new 
e-mail addresses in order to join the study. During the 4- 
month recruitment period, HMC staff used a multi- 
faceted approach that included letters of invitation mailed 
to all eligible patients with known addresses, active 
recruitment of a subset of eligible patients by telephone 
and in person recruitment of all eligible patients who 
came into the primary care clinics for a regularly sched- 
uled clinic visit. Eligible patients who were approached 
about the study in the clinics were given an information 
flyer about the study and were invited to enroll after their 
clinic appointment or to join a bi-weekly group enrollment 
session. To enroll, HMC patients provided their email 
addresses, e-signed the study consent form, and then com- 
pleted the online pre-intervention survey. 

OpenNotes: the intervention 

Prior to OpenNotes, patients registered on the multifunc- 
tional GHS or BIDMC patient portals already had online 
access to their problem lists, medication records, and 
laboratory and radiology reports. In addition, the portals 
provided appointment scheduling and secure e-mail mes- 
saging among patients and their doctors and health care 
teams. As participants in the OpenNotes project, the 
PCP's at BIDMC or GHS could selectively exclude their 
patients, as described above, but they would be allowing 
all of their other eligible patients access to their visit notes 
if they were registered portal users at the time the study 
began in Spring, 2010. 

At BIDMC and GHS, the OpenNotes intervention con- 
sisted of a simple addition to the existing menu of acces- 
sible records. Following a scheduled office visit, PCPs 
would record and sign their visit notes, at which point 
their patient would receive an automatically-generated e- 
mail invitation to read the visit note. Two weeks prior to 
a next scheduled visit to their PCP, patients also received 
an e-mail message suggesting that they review prior 
"OpenNotes" before coming to see their doctor. At 
HMC, patients did not have prior online access to their 
health records; therefore the intervention included new 
access not only to the visit notes, but also to other sec- 



tions of their EHR including laboratory and radiology 
reports. Unlike the portals at BIDMC and GHS, the 
HMC portal was not interactive; patients could only view 
their records but could not schedule appointments or 
send messages to the healthcare team. 

Surveys of doctors and patients 

We are using several data sources in the multi-faceted eva- 
luation of OpenNotes, but the principal sources of infor- 
mation used to assess the study's primary outcomes are 
PCP and patient surveys conducted prior to, and then 
immediately following the one-year OpenNotes interven- 
tion. All surveys were conducted online using the web- 
based program, SurveyGizmo. Both intervention and non- 
intervention PCP's and their patients from the BIDMC 
and GHS were provided links to the surveys via survey 
invitation messages sent through the portals. Each respon- 
dent was given a unique study ID number assigned by the 
information technology (IT) staff of each health system. 
Survey data were downloaded from the password-pro- 
tected SurveyGizmo databases into password protected 
files on the computer network at the study coordinating 
center at the BIDMC. At HMC, patients completed their 
surveys at the time of enrollment which took place in their 
HMC clinic. 

Electronic databases 

In addition to the surveys, IT staff of each site assembled 
electronic data from multiple existing sources at the 3 
sites, including the portal tracking systems, administrative 
databases and health care utilization databases. Using the 
unique study ID assigned to both intervention and non- 
intervention PCP's and patients at each site, we collected 
demographic and health characteristics, information about 
portal use, and health care utilization data for the year 
prior to the start of the intervention and for the one-year 
during which the visit notes were made available to inter- 
vention patients. 
Patient characteristics 

We used administrative data sources to obtain demo- 
graphic information (age and sex) about PCPs and patients 
at each of the 3 sites. 
Portal use data 

We assessed portal use at each site through their elec- 
tronic tracking systems. At BIDMC, HMC and GHS, the 
systems recorded the time and date of each patient por- 
tal login, each click into each section of the portal, 
including the OpenNotes section, and, with the excep- 
tion of HMC, each message sent or received from 
within the portal. During the year of the intervention, 
we determined the total number of times patients 
entered the new OpenNotes sections of their portal 
records. 
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Health care utilization 

Patient utilization measures, determined from billing 
records, included counts of visits to PCP's and their 
practices, hospitalizations, and emergency room visits. 

Results 

Rates of PCP participation in the OpenNotes intervention 
varied widely among the 3 study sites. At GHS, 27 (19%) 
out of 145 invited PCPs enrolled; at BIDMC, 42 (66%) out 
of 64 invited PCPs enrolled; at HMC, 45 (87%) out of 52 
invited PCPs enrolled in the intervention. Invited PCPs 
who declined participation at BIDMC and GHS were 
included in the control group of non-participating PCPs 
(GHS, n = 118; BIDMC, n = 22). The recruitment and 
enrollment of PCPs and their patients at the 3 participat- 
ing sites are shown in Figure 2A-C. 

From among their patient panels, intervention PCPs at 
BIDMC and GHS excluded 158 and 139 patients from 
participating, respectively, or 1.3% and 1.4% of eligible 
patients. At HMC, the entire panel of adult patients mana- 
ged by each intervention PCP was initially eligible for the 
study, and the PCPs excluded 1,006, or 32% of their eligi- 
ble patients. Though we did not collect reasons for exclu- 
sion, the HMC patient population includes many 
individuals with psychiatric diagnoses, current substance 
abuse, or other severe medical conditions (eg. bone mar- 
row transplant). 

At the BIDMC, 11,740 patients (39%) from the panels of 
participating PCPs were registered portal users and eligible 
to participate in the intervention. Only 15 patients (0.1%) 
requested to not be included in the intervention after 
receiving their notification letter about OpenNotes and 
their invitation to complete the survey. 4,545 intervention 
patients (39% of those invited) submitted the optional 
online survey prior to the start of the intervention. On the 
panels of the non-participating PCPs, 6,631 patients were 
registered on the portal and invited to complete the online 
survey; 2,701 (41%) submitted the survey. 

Within GHS, of the nearly 45,000 patients on the panels 
of participating doctors, 10,686 patients (24%) were regis- 
tered on the MyGeisinger portal. Of these, 4,226 (40%) 
responded to the online survey. Among the PCPs who did 
not participate in the intervention, 61,492 patients were 
registered on the portal and invited to participate in the 
survey. Of these, 26,528 patients (43%) submitted the 
online survey. 

At HMC, patient recruitment was individualized because 
there was no previous web portal available to patients. For 
the initial recruitment, invitation letters were mailed to 
2,163 eligible patients; however, 179 patients (8%) did not 
have a viable mailing address or their letters were returned 
undeliverable. The second step in the recruitment process 
involved telephoning patients from the PCPs' panel lists, 
or directly approaching patients in the clinic before their 



clinical appointment. Of 434 patients contacted by phone 
or approached in the clinics, 155 were ineligible primarily 
due to language barriers (n = 34) or lack of an email 
address (n = 113). Eligible and interested patients were 
asked to complete the survey online as part of the study 
enrollment. Five patients (2% of enrollees) experienced 
technical problems submitting their surveys which, as a 
result, were not recorded. There were 272 HMC patients 
who completed the online survey. 

Characteristics of PCPs 

Doctors who participated were somewhat younger than 
non-participating PCPs at BIDMC and HMC (Table 2); 
there were higher proportions of men among the partici- 
pating PCPs; and participating PCPs generally had smaller 
panels, compared to non-participating PCPs. However, 
none of the observed differences between groups were sta- 
tistically significant. 

Characteristics of patients 

On average, both participating and non-participating 
patients were approximately aged 50 years at each of the 3 
sites (Table 3). Approximately 60% were female in the 
Boston and Pennsylvania sites, with a slightly higher per- 
centage of women in the non-participating patient groups. 
In Seatde, 24% of participating patients were women. 

Discussion 

To our knowledge, this is the first extensive evaluation of a 
large-scale implementation of online patient access to doc- 
tors' visit notes through patient Internet portals. Our 
recruitment process demonstrates that substantial num- 
bers of primary care providers are willing to engage in a 
year-long trial of this potentially disruptive intervention. 
Large numbers of primary care patients have had access to 
sections of their health records online for several years at 
the BIDMC and GHS. Although the addition of the office 
visit notes to these readily accessible online records 
seemed like a relatively minor change, many doctors 
voiced concerns in advance of the trial and even opposi- 
tion to this next step in the evolution of transparency in 
healthcare. 

Moving beyond small scale and qualitative studies of 
the impact of patient access to medical records, we 
designed the study to involve 3 large institutions with 
distinctly different patient populations. We intended to 
enroll as many PCPs and patients as possible and to cap- 
ture as much quantitative and qualitative information as 
possible in our evaluation. Thus, in planning the evalua- 
tion and in giving careful consideration to our choice of 
research design and methods, we sought to collect new 
information and to maximize use of existing data sources. 
We used an online survey to address our key hypothesis 
about whether or not PCPs and patients would want to 
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Eligible Primary Care 
Physicians (PCP) N=64 



Declined to Participate 
N=22 (34%) 



Patients on panels 
N=19,851 



PCPs consented to 
Intervention N=42 (66%) 



1 PCP withdrew before 
intervention 



Patients on panels 
N=30,885 



Patients not on portal 
N=13,220 



Non-intervention Patients 
(on portal) 
N=6631 (33%) 



Excluded by PCP N = 158 
Patients not on portal N=18,987 



Intervention Patients 
N = ll,740 (39%) 



Declined participation 
N=15 (0.1%) 



Unreachable n=3 (0%) 
Declined Survey N=42 (1%) 
Non-Respondents N=3,885 (58%) 



Survey respondents 
N=2,701 (41%) 



Survey respondents 
N=4,545 (39%) 



Unreachable N = 150 (1%) 
Declined Survey N=2 (0%) 
Non-Respondents N=7,028 (60%) 



Eligible Physicians N=145 



Declined to participate 
N=118 (81%) 



Consented and enrolled 
N=27 (19%) 





Patients on panels 
N= 240,980 


Patients not on portal 
N= 179,488 







Non-intervention Patients 
(on portal) N=61,492 (26%) 



Patients on panels 
N =44,900 








Excluded by PCP N=139 
Patients not on portal N=34,075 


Interven 
N = 10, 


tion Patients 
86 (24%) 





Survey Non-Respondents 
N=34,964 (57%) 



Survey Respondents 
N=26,528 (43%) 



Survey Non-Respondents 
N=6,460(60%) 



Survey Respondents 
N=4,226 (40%) 



c. 



Eligible PCPs: N=52 



□ L 



Declined 3 (6%) 



Consented 49 (91%) 



PCPs Completed 
Survey: 2 



No Panel Review: 
Non-Participating PCPs: 4 



I 



Reviewed Panel: 
Participating PCPs: 45 



PCPs Completed 
Survey: 40 



Total patients on PCP panels 
N=3,186 



X 



Ineligible: N= 1,023 (32%) 
1,006 excluded by physician 
17 bone marrow transplant patients 



Initially Eligible Patients 
N=2,163 (68%) 



Unreachable: 161 
Ineligible: 155 
Declined: 81 



Letters mailed to 
2,163 Patients 



Called by phone or 
approached in clinic: 
N=434 



179 Letters 
undeliverable 



Intervention Patients 
N= 277 



Survey non-respondents 
N^5 (2%) 



Survey respondents 
N=272 (98%) 



Figure 2 (A) BIDMC PCP and Patient Recruitment and Enrollment, (B) GHS PCP and Patient Recruitment and Enrollment, (C) HMC PCP 
and Patient Recruitment and Enrollment 



continue OpenNotes at the conclusion of the trial. To 
explore this further, the survey included questions asses- 
sing attitudes and perceptions about the impact of Open- 
Notes. We made use of both administrative data and data 



from portal tracking systems to address our secondary 
hypotheses, to cross-validate survey responses from PCPs 
and patients with actual portal use before and during the 
intervention period. 
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Table 2 Characteristics of participating and non-participating primary care physicians (PCPs) according to study site* 





BIDMC PCPs 


GHS PCPs 


HMC PCPs 


Characteristics 


Participating (n 
= 42) 


Non-participating (n 

= 22) 


Participating (n 
= 27) 


Non-participating (n 
= 118) 


Participating (N 
= 45) 


Non-participating 
(N = 4) 


Age (mean ± S. 
D.) 


47.3 ± 9.6 


50.8 ± 6.7 


49.7 ± 8.2 


49.9 ± 9.4 


42.5 ± 6.9 


49.8 ± 5.4 


Sex (female) 


20 (47.6%) 


11 (50.0%) 


6 (22.2%) 


38 (32.2%) 


26 (57.8%) 


3 (75.0%) 


Panel size: 














< 500 


1 8 (42.9%) 


7 (31.8%) 


4 (14.8%) 


9 (7.6%) 


45 (100%) 


4 (100%) 


500-999 


14 (33.3%) 


7 (31.8%) 


5 (18.5%) 


12 (10.2%) 


0 


0 


1000-1999 


8 (19.0%) 


6 (27.3%) 


7 (25.9%) 


42 (35.6%) 


0 


0 


> 2000 


2 (4.8%) 


2 (9.1%) 


11 (40.7%) 


55 (46.6%) 


0 


0 



* Differences between participating and non-participating PCPs tested using t-tests for continuous measures and Chi-square tests for categorical measures. No 
significant differences were observed between participating and non-participating PCPs at any of the 3 sites 



Table 3 Characteristics* of participating and non-participating patients according to study site 

Characteristics BIDMC Patients GHS Patients HMC Patients 

Participating Non-participating Participating Non-participating Participating 

Age (mean ± S.D.) 49.4 ±13.7 51.3 ±14.3 49.9 ± 15.6 49.4 ±15.3 48.9 ±11.0 



Sex (% female) 57.4% 59.6% 

^Information derived from administrative records of each site 

Our selection of study sites was intended to create a 
diverse pool of PCP practices and their patients. 
Although the majority of PCPs from the Boston (66%) 
and Seattle (87%) sites agreed to join the intervention, 
in Pennsylvania, a much smaller proportion of providers 
(19%) from these rural practices enrolled. On the other 
hand, panel sizes in Pennsylvania were generally much 
larger than the other 2 academic-affiliated institutions, 
and large numbers of patient from GHS (more than 
10,000) were thereby enrolled in the intervention. Unlike 
the other sites, recruitment of the geographically dis- 
persed GHS providers relied more on written communi- 
cation than in-person discussions and presentations 
about the study, a factor that may have contributed to 
lower PCP participation. 

In conclusion, many primary care physicians were 
willing to participate in this new intervention despite 
concerns of a potential burden to their practices related 
to patient inquiries about their notes. This positive 
response attests to the feasibility of conducting a large 
scale, multisite study of the impact of an intervention 
using electronic health records and Internet patient por- 
tals. Extensive electronic data resources will make it 
possible to conduct a thorough multi-faceted evaluation 
of PCP and patient level factors to determine the impact 
of giving patients online access to their physician's office 
visit notes. In addition, the availability of online survey 
technology enabled us to collect extensive information 
about attitudes and expectations before notes were 
made readily available, and about actual experiences 



58.0% 61.8% 23.7% 



after using such notes for at least one year. Results from 
this evaluation will inform providers, policy makers, and 
patients who contemplate such changes at a time of 
exploding interest in transparency, patient safety, and 
improving the quality of care. 
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